\begin{abstract}
%\boldmath
MapReduce systems face enormous challenges due to increasing growth, 
diversity, and consolidation of the data and computation involved.
Provisioning, configuring, and managing large-scale MapReduce clusters
require realistic, workload-specific performance insights 
that existing MapReduce benchmarks are ill-equipped to supply. 

In this paper, we build the case for going beyond benchmarks
for MapReduce performance evaluations. We analyze and compare two
production MapReduce traces to develop a vocabulary for describing
MapReduce workloads. We show that existing benchmarks
fail to capture rich workload characteristics observed in traces, and propose a framework
to synthesize and execute representative workloads. 
We demonstrate that performance evaluations using realistic workloads
gives cluster operator new ways to identify workload-specific 
resource bottlenecks, and workload-specific choice of MapReduce 
task schedulers. 

We expect that once available, workload suites
would allow cluster operators to accomplish previously challenging
tasks beyond what we can now imagine, thus serving as a useful tool 
to help design and manage MapReduce systems.
\end{abstract}
